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Evaluating the Impact of A Staff Development Program: 
ANALYSIS OF DOCUMENTATION IN PERSONNEL RECORDS 



Investigators: Nicholas DeHgio, Naomi Zigmond and Paul LeMahieu 
Institute for Practice & Research in Education 
University of Pittsburgh 



Intrftduction 

Much has been written oa the essential qualities of effective building principals. 
Effective principals have clear, inform^ visions of what they want their schools to become. 
They translate these visions into goals for their schools and expectations for their teachers, 
students and other administrators. They establish school climates that support progress towards 
these goals and expectations. They continuously monitor progress and intervene in a supportive 
and corrective manner when this seems necessary (Cotton and Savard , 1980; Persell and 
Crookson, 1982 and Stow and Manatt, 1982). Because many principals need to leam how to do 
these things, i.e., how to be more effective, staff development programs initiatwl by school 
districts often include a set of activities aimed specifically at princip^. 

Th«e has been a great deal of emphasis placed in recent years on staff development, 
but few studies of its impact on performance. Those few studies which have been conducted 
have focused on teacher behaviors and student achievement that may have changed as a function 
of a district wide school improvement effort Although principals always take part in the staff 
development program, and are seen as essential players in any school improvement effort, there 
have been very few evaluations of the impact tf a staff develiroment program on the poformance 
of principals of their assigned tasks. The literature on the effective principal focuses on the role 
of the principal a. ui instructional leader and on the consultative role the principal can play in 
assisting teachers to improve their performance. But principals are required, by state regulation, 
to evaluate the teachers in their buildings and to provide documentation when such evSuations 
earn a teacher a less than Satisfactory rating. This smdy explores the usefulness of a unique set 
of data, the documentation provided by principals to accompany their teacher evaluations, as 
indicators of the impact of an inservice program on the i^ormance of principals of their jobs. 
The study is useful not only in its contribution to the Utcramre on the effectiveness of inservice 
training of administrators, but also for plowing new ground in evaluation strategy. Therefore the 
purpose of this research was to evaluate the impact of one such staff development program on the 
performance of principals in one of their most important tasks: the annual evaluation of teachers. 
The objective was to determine whether the effects of the staff development training could be 
seen in a change in the quality of documentation provided by principals in their annual review of 
leachCTs whom they found to be Unsatisfactory . 



Perspective 

One of the responsibilities of the building principal has always been to evaluate 
teachers. Staff development efforts designed to help principals to be more effective often focus 
on increasing their competence in observing instruction, diagnosing teaching strategies and 
improving classroom methodologies or instructional effectiveness. Principals '^am that 
infrequent summative evaluations of teachers arc often superficial and not helpful in improving 
teacher performance; they leam to implement formative, participatory, diagnostic evaluations of 
clasOT)om teaching so that teacher and principal can get at real needs and develop real processes 
for change and growth. They leam models of teaching and how to recognize these models in 
classroom observations. They leam that data collection and documentation are important to the 
process of formative evaluation - and Jheir observation notes form the basis for discussions with 
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teachers about teaching. But, training in forma;1ve evaluation does not remove from the principal 
the responsibility for summative evaluation. True, the drop in pupil enrollment throughout many 
districts has reduced the number of probationary teachers and increased the percentage of older, 
tenured faculty in schools. Yet there arc teachers who arc still incompetent. Once principals are 
trained in the formative evaluation of classitx>m instructional processes, they should be able to 
recognize these teachers more easily, provide both helpful feedback to them in a concerted effon 
to improve their competence, and document their inadequacies to sustain a judgement of 
incompetence (DeBcvoize, 1984; Howell, 1985; Lippitt, Langseth and Mossop, 1985; and 
Mangieri and Arnn, 1985). 

This study was designol to evaluate the impact of a staff development program which 
focused on teaching principals instructional leadership skills through an analysis of the 
documentation they provided in dossiers of teachers they rated as Unsatisfactory . If principals 
were getting better in instructional leadership and learning what was being taught them in the staff 
development program, it was likely to be reflected in the language used to describe teacher 
instructional and mana^ment behaviors, in the quality of the observation notes, and in the nature 
of the recommendations for improvement. Further, imprcvements in instructional leadership 
might lead to an increase in the number of observations undertaken by principals as they try to 
influence teacher behavior, and the documentation of these observations might also reveal a 
consistent focus over time. By reviewing documentation in personnel files (dossiers) of 
Unsatisfactory teachers for a period of yean before the implementation of the staff development 
program for principals, then for several years after the inservice trainmg had been underway, the 
researchers sought to identify whether the impact of the training could be seen in changes in the 
quality and quantity of documentation provided by principals to support their ratings. Therefore 
the issue addressed in this investigation was whether the impact of the a staff development 
program on Piincipal's behavior can be seen in changes in the numbers rated Unsatisfactory or 
Below Average over the last ten years, or in changes in the quality and quantity of the 
documentation provided bv Principals to support an Unsatisfactory raring. 



Pilot Investigation 

Two sources of data were made available to the investigators to explore the feasibility 
of utilizing data on the numbers of teachers rated as Unsatisfactory each year as well as the 
documentation available in Personnel files of teachers rated Unsatisfactory as evidence of the 
impact of training on Principals. First, Personnel staff provided rough estimates of the numbers 
of Unsatisfactory and Below Average ratings of teachers per school year from 1977-78 to 
1986-87. Second, seven records of teachers who were given Unsatisfactory ratings, were drawn 
at random, two each from 1976-77 and 1980-81 and three from 1985-86. These records were 
purged of all identifying information by clerical staff (a very time consuming task) in the 
Personnel Office and made available to the investigators. 



Preliminary Impressions 

1) The numbers of Unsatisfactory ratings have remained fairly stable over the last 10 
years (approximately 20 per year), although there appears to have been a substantial increase in 
Ihe number of Below Average ratings beginning in 1983-84 (from an average of 6-10 per year 
fi'om 78 to '83, to an average of 14-16 per year in '84 to '86). 

2) Principals are not doing more observations of teachers who end up with 
Unsatisfactory ratings than they did before, but the more recent documentation is more detailed, 
makes use of staff development terminology to describe instructional problems, and is much 
richer in suggestions for improvement. 

From the preliminary examination the following questions were generated and they 
served as the guide for the major study. 
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What percentage of the teaching force has been rated Unsatisfactory each year? 

How many of t*^e Unsatisfactories are challenged in hearings each year? 

Has that changed over ten years? 

What proportion of Unsatisfactories end in dismissals? Has it changed over ten years? 

Has the focus of the documentation changed over the years from managerial to 
instructional? 

Have the numbers of observations done by Principals to document an Unsatisfactory 
teacher change over ten years? 

Is the instructional documentation critical, or ancillary, to the development of an 
Unsatisfactory dossier? 

Have the rates of favorably held grievances changed over the past ten years? 

Have the bases for grievances and arbitration changwi? 



Description of the Staff Development Program 

Staff development in this district was organized to introduce a shared professional 
language and a system-wide approach to classroom instruction, improve classroom instruction, 
support instructional leadership, develop a process for personnel evaluation, and stimulate 
professional growth (Mensingcr, 1986). 

All teachers and administrators in the school district participated in the staff 
development program. During the first year principals and supervisors received 30 hours of 
training that included an ''update in ^e principles of effective teaching, observation skills, 
analysis and feedback skills. During and after the training period idminisirators observed, held 
conferences and taught staff development concepts to teachers" (Stoeckingcr, 1984). 

In 1982-83 the principals and supervisors, with the help of a small group of teachers, 
trained other teachers. By the end of the 1982-83 school year, all personnel had been exposed to 
the staff development concepts. The focus was on "knowledge, skills, and attitudes that are 
prerequisite to effa:tive educational leadership" (Wallace et al, 1984). 

The third phase, a center for secondary school teachers, provided a clinical experience 
for all high school teachers in the district The purpose was to give each teacher the opportunity 
"(1) to observe instructional activities in a real setting; (2) to practice new skills and techniques; 
(3) to receive fredback on that practice; (4) to translate theory into practice; and (5; to recieve an 
update m their specific subject matter areas, latest research finding in effective teaching 
technology, and appropriate areas in psychology" (Wallace et al, 1984). Centers at the middle 
and elementary schools were to follow the development of the secondary school center. District 
personnel are presently engaged in center activities. 



Methods and Data Source 

The data source for this study consisted of a random sample (44 of 109) of personnel 
files of teachers who have been rated as Unsatisfactory from a school district in Westein 
Pennsylvania. (Personnel files of teachers who were rated as Satisfactory were not reviewed 
because principals are not required to file documentation to support the Satisfactory rating). A 
staff development program for principals was initiated in this school district in 1981-82 as part of 
a concerted long term school improvement effort. Beginning with the 1977-78 school year (five 
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years before the implementation of the staff development program) and continuing to 1987-88 
(six years after the initiation of the principal's staff development program), 50% or no more than 
five personnel file records were selected for review from each year. Records were selected at 
random by the principal investigator firom a list of all those rated unsatisfactory in each of 1 1 
school years. The researcher reviewed the documentation in 44 personnel file folders that 
included all the records of the teachers as well as tiic documentation of the Unsatisfactory rating. 

The principal investigator made notations on the following items: employment history 
of each teacher reviewed; number of observations documented in the dossier and the 
administrative role of tiie document^- for each d5servation; instructional and managerial problems 
noted in each observation and frequency of notations of each type of instructional or managerial 
problem across all observations; number and types of recommendations made for improving 
teacher performance for each observation; presence of consistent 'themes' across any two 
consecutive observations in documentation of teacher performance or in recommendations made; 
presence of staff development 'terminology' in the documentation; and presence of 'critical 
incidents' that warrent an Unsatisfactory raring even without additional documentation on 
instructional effectiveness. Then the researcher organized the information from his notes using 
the above categories and developed a summary for each of eleven years. This summary 
described the nature of the Unsatisfaaory ratings by years. 

Data ftxMn the 44 personnel files were analyzed to determine whether the focus of the 
documentation by principals had changed over the eleven year period and whether die timing of 
these changes coincided with the implementation of the staff development program; whether the 
numbers of observations done by principals to document Unsatisfactory ratings had changed 
over the eleven year period; whether the quantity and substance of recommendations had 
changed; whether there were more consistent 'themes' running through consecutive 
documentations in more recent dossiers; and, whether the instructional documentation provided 
in the dossiers was essential to developing the case for each Unsatisfactory rating or was actually 
ancillary because of the presence of critical incidents that in and of themselves warranted 
xtieUnsatisfactory ratings. 



Findings, 

1. The number of teachers rated unsatisfactory increased from 0.21% of the teaching force 
during 1972-82 to 0.47% of the teaching force during 1983-88; in fact, the numbers have 
actually doubled (see Figure 1). 

2. Of the teachei-s who were rated unsatisfactory during 1978-82, 30% are still teaching in the 
district. Of the teachers who were rated unsatisfactory during 1983-87, 46% are still 
teaching. This difference may be due to the longer period of rime that the teachers in the 
1978-82 category have had to reach or approach retirement age (see Table 1). 

3. The number of teachers rated unsatisfactory who resigned or retired between 1978-82 (11) 
and 1983-88 (11) remained the same. However, the teachers who resigned or retired in 
the 1983-88 category did so much closer to the time of the unsatisfactory rating (i.e., 
within one or two years). This suggests that, more recentiy, a greater effort has been made 
to move unsatisfactory teachers out of the system during this time period (see Table 2). 

4. Most teachers do not grieve the unsatisfactory ratings that they receive. Only 15% of 
grievances of unsatisfactory ratings filed between 1983 and 1988 resulted in a change in 
rating as compared to 60% for grievances of unsatisfactory ratings that occurred from 1976 
through 1982 (see Table 3). 

5. The effect noted above does not extend to below average ratings as well. Eighty-eight 
percent of grievances of below average ratings resulted in a changed rating during the 
1983-88 period as compared to 25% for grievance of below average ratings in the 1976-82 
period (see Table 3). 
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6. There are the same number of observation/conference forms in the personnel folders now 
as before (average 6.1 per teacher). However, the number of teachers rated unsatisfactory 
in which more than three people are used as sources of documentation has increased from 
25% to 50%. The number of personnel dossiers with observations/conferences only by 
the principal has decreased from 38% to 20% (see Table 4), 

7. The percent of teachers whose unsatisfactory ratmg was the result of only a smgle critical 
incident has decreased from 19% over the period 1977-82 to eight percent between 
1983-88 (see Table 4). 

8. The number of schools in which teachers rated unsatisfactory have serving in the five years 
prior to the unsatisfactory rating has decreased. Over the period 1977-82 the average 
number was 2.6; since that time the average has been 2.2. The percent of teachers rated 
unsatisfactory serving in three or more schools over a five year period has decreased from 
38% to 25%. Moreover the years 1986-87 and 1987-88, the only years for which there is 
a full five years following the increase in evaluation efforts, shows a drop (from 2.6 to 
1.4) in the average number of different schools, (see Table 5). 

9. Of the 109 teachers rated unsatisfactory from 1977-1988, seven teachers received a second 
unsatisfactory rating. Five of these seven ratings have occurred since 1982-83 and six of 
the seven teachers received second unsatisfactory ratings the next year. None of the seven 
teachers is still employed. 

10. The average age of the teachers rated unsatisfactory was 42.5 years for 1977-82 and 46.6 
years for 1983-87. Twelve of the 44 teachers reviewed were below 40 years of age and 32 
wCTC 40 years of age or older (see Table 1 and Figure 2). 

11. The number of PRISM terms used in the documentation of unsatisfactory ratings has 
changed over time, showing a significant increase in 1983-84 and remaining high (see 
Figure 4). 

12. The distribution of teachers rated unsatisfactory by organizational level of service suggests 
a fairly even distribution by level within regular education, but perhaps a 
disproportionately small number of special education teachers have been rated 
unsatisfactory (see Figure 3), 



Sun^mary 

Analyses of the results of this study demonstraf that the quality of the documentation has 
improved over the years and that principals are doing more observations, that the more recent 
documentation is more detailed, makes use of the instructional terminology taught to principals 
in the staff development program, and is much richer in suggestions for improvement, since the 
beginning of the district's staff development effort. 

In addition, the analyses of employment history revealed that a single critical incident often 
triggered a flurry of observations and conferences by the principal, vice principal, dean or subject 
area supervisors to gather supporting evidence for an Unsatisfactory rating. Nearly 90% of aJI 
teachers receiving Unsatisfactory ratings had a history of being transferred to other buildings or 
other grade levels within the district Many of these same teachers rtztiv^ Below Average 
ratings at some point in their career and had received one or more reprimands for some infraction 
of rules or regulations, 

Nearly all the records reviewed showed classroom organization as the single most 
prominent weakness of the teachers rated Unsatisfactory. Poor planning and methodology were 
always seen by the rater as the reason for Unsatisfactory management and organization. Only 
one eacher was cited for lack of content knowledge but this was added to the majorproblem of 
management and control of the classroom and children. 
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Figure 1 

Unsatisfactory and Below Average Ratings for 

Ten Year Period 




1978 1979 1980 1981 1982 1905 1984 1985 1986 1987 1988 

School Year 

^ Unsatisfactory Ratings 
■ Below Average Ratings 

— g — 



DcRgio/Zigmond/LeMahieu 

Table 1 

Disposition of Professional Employees Receiving Unsatisfactory Ratings 1977-78 thru 1987-88 



School #Rec Average # Still Number Number Number 

Year Unsai lOUZl&ysi ESWL RglOT TCTTOP^^e 



1977-78 


9 


405 


0 


4 


3 


2 


1978-79 


10 


35.8 


3 


3 


3 


1 




A 






Km 


0 


1 


1980-81 


4 


48.5 


0 


2 


2 


0 


1981-82 


Jl 




± 


i 


i 


a 


subtotals 


33 


42.5 


10 


10 


9 


4 












27 2% 




1982-83* 


7 


46.0 


4 


2 


1 


0 


1983-84* 


13 


45.0 


7 


4 


0 


2 


1984-85 


8 


48.0 


0 


5 


3 


0 


1985-86 


12 


43.4 


6 


2 


4 


0 


1986-87 


IQ 






i 




Q 


subtotals 


50 


46.6 


23 


14 


11 


"J 








46% 


28% 


22% 


4% 


Totals 


83 


44.5 


33 


24 


20 


6 








39.7% 


28.9% 


24% 


7.1 


♦Note: School nurse not counted; no record of disposition. 








1987-88** 


26 


49.4 


21 


1 


I 


0 



**Note: 1987-88 distorts differences because no one on this list was on last years list. 
Therefore, all of them have one nrore year technically/legally. (Numbers do not add up to 26 - 3 
teachers took sabbatical leaves). 
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Table 2 

Disposition of Unsatisfactory Rated Teachers by Year When They Resigned or Retired 



Year 
Reed 

Unsat Still 

Essm 1978 im mo im im i98? \m \m iw \m smpiQv 

1978 6/78 

1978 1/79 

1978 12/85 

1978 6/78 

1979 V 

1979 V 

1979 6/84 

1979 6/79 

1979 2/86 

1980 V 

1980 V 

1981 V 

1981 6/81 

1982 V 

1982 6/85 

1982 V 

1983 V 

1983 V 

1983 V 

1983 V 

1984 V 

1984 \ 

1984 \ 

1984 6/84 

1984 12/84 

1985 11/85 
1985 6/85 
1985 4/85 

1985 1/86 

1986 — V 

1986 \ 

1986 6/86 

1986 V 

1986 6/87 

1987 V 

1987 6/87 

1987 6/88 

1987 6/87 

1987 V 

1988 > V 

1988 V 

1988 V 

1988 . 8/88 

1988 \ 



1 



Table 3 

Disposition of Actions; Unsatisfactories and Below Average, 1977-78 through 1986-87 



Time Period 



Rating 



Percent of Grievance 
Number Percent Upheld 
Grieved Grieved (Rating Changed) 



1976-82 



Unsatisfactory - 33 
Below Average -22 



5 
4 



15% 
18% 



60% (3/5) 
25% (1/4) 



1983-88 



Unsatisfactory - 50 
Below Average -48 



13 
8 



26% 
17% 



15% (2/13) 
88% (7/8) 



Table 4 

Number of Different People Observing and Conferencing With Unsatisfactory Rated Teachers 













Number 




School 








Assoc 


Obs/Conf 


Cnucal 


Year 


Prin. 






Dir JQit. 


on Hie 




1977-78 


X 








0 


X 




X 


X 






6 






X 








S 






X 


X 






13 




1978-79 


X 




X 




11 






X 


X 






2 






]?C 


X 


• 




2 






X 


X 




X 


7 






X 








0 


X 


1979-80 


X 








6 








X 






6 




1980-81 


X 


X 




X 


13 






X 


X 


X 




13 




1981-82 


X 








1 






X 


X 


X 




13 






X 








0 


X 


1982-83 


X 




X 




5 






X 


X 




X 


8 






X 








0 


X 




X 








0 


X 


1983-84 


X 








0 


X 




X 








9 






X 




X 




11 






X 


X 


X 




6 






X 


X 




X 


5 




1984-85 


X 








0 


X 




X 


X 


X 




10 






X 


X 






11 






X 


X 


X 


X 


6 




1985-86 


X 


X 


X 




6 






X 


X 


X 




7 






X 








9 






X 




X 




16 






X 


X 


X 




U 





Table 4 (Cont) 

Numbw of Different Pedple Observing and Conferencing With Unsatisfactory Rated Teachers 



Number 

School Dean/ Assoc Obs/Conf Critical 

JCS3L Emi YPrin Ds: J2iL ooBlc iDosism 

1986- 87 XXX 3 

XXX 3 

XXX 6 

XX 6 

XX 4 

1987- 88 X 6 

X X 8 

XXX 3 

XX 5 

XXX 7 
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Tabic 5 



Number of Different Schools Unsatisfactory Teachers Have Been in Within Previous Five Years of 
the Unsatisfactory Rating 



1977-78 



1972-78 
RrstYcar 

1971- 76 

1972- 78 



Number of Schools AS£ 



2 
1 
2 



Average 



22 



50 
24 
57 
11 

40.5 



1978-79 



1973- 79 
1979 
1979 

1974- 79 
1973-79 



Average 



3 
1 
1 
1 

1 
1.8 



34 
25 
22 
51 

35.8 



1979-80 



1976-80 
1975-80 



Average 



4 
Z 



52 
U 

40 



1980-81 



19'/6-81 
1976-81 



Average 



1 

2 

1.5 



56 
48.5 



1981-82 



1979-82 

1976- 82 

1977- 82 



Average 



4 
3 
1 



31 
60 

47.6 



1982-83 



1978-83 
1977-83 
1977-83 
1977-83 



2 
1 
2 
1 



43 
54 
55 



Average 1.75 



46 



16 



Table-S-CCont) 

Number of Different Schools Unsansfactoiy Teachers Have Been in Within Previous Five Years of 
the Unsatisfact(^ Rating 



1983-84 



School Year 

1978-84 
1978-84 
1978-84 
1978-84 
1978-84 



Average 



Number of Schools 

3 
2 
3 
4 
i 

2.6 



Ass 

46 
47 
51 
40 
il 

45 



1984-85 



1980-85 
1980-85 
1980-85 
1980-85 



Average 



2 
2 
2 
2 



53 
33 
47 

m 

48 



1985-86 



1981-86 
1981-86 
1981-86 
1981-86 
1981-86 



Average 



4 

1 
4 
6 
1 



35 
60 
55 
29 
i& 

43.4 



1986-87 



1982-87 
1982-87 
1982-87 
1982-87 
1982-87 



1 
2 
1 
1 

a 



50 
54 
60 
51 



Average 



1.6 



51 



1987-88 1983-88 2 42 

1983-88 1 40 

1983-88 1 63 

1983-88 1 56 

1983-88 i Mi 

Aveiagc 1.2 49.4 




Figure 2 

Ages of Teachers Rated Unsatisfactory 

(n « 44) 
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Figure 4 
Use of PRISM Terms/Phrases 
(Average by Years) 




•Only two teachers were levicwcd this year, 
only one file had any (4) PRISM t^ins. 

♦•Obvious increase occurred from *982-83/ 1983-84 
^ Use of icims remained high through 1987-88. 
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Figures 

Distribution of Unsatisfactory Ratings According 
to Organizational Level 
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